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Abstract: We present test results and characterization of a data transmission system based on a 
last generation FPGA and a conmiercial QSFP+ (Quad Small Form Pluggable +) module. QSFP+ 
standard defines a hot-pluggable transceiver available in copper or optical cable assemblies for 
an aggregated bandwidth of up to 40 Gbps. We implemented a complete testbench based on a 
commercial development card mounting an Altera Stratix IV FPGA with 24 serial transceivers at 
8.5 Gbps, together with a custom mezzanine hosting three QSFP+ modules. We present test results 
and signal integrity measurements up to an aggregated bandwidth of 12 Gbps. 
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1. Introduction 

High-speed data transfer plays a central role in trigger and data acquisition systems for present and 
future (sLHC, ILC,vCLIC, rare kaon decays) particle and astroparticle physics experiments. 

We present test results and characterization of a data transmission system aimed at very high 
throughput applications based on last generation FPGA-embedded serial transceivers and commer- 
cial QSFP+ (Quad Small Form Pluggable +) modules. 

This data transmission system was originally conceived for a commercial PC cluster for lattice 
QCD and other computing intensive numerical algorithms [IID^Ol^ED- However it can also be of 
interest for future particle and astroparticle data acquisition systems given its high bandwidth, low 
power and low cost characteristics. 

1.1 High-speed FPGA serial transceivers 

High-speed FPGA-embedded serial transceivers technology greatly evolved in the last 10 years and 
is now mature and reliable with multi Gbps transceivers easily available on commercial devices. 
In particular we focus on an Altera Stratix IV GX FPGA 0] with embedded full duplex serial 
transceivers with a data rate from 600 Mbps to 8.5 Gbps, 8B10B encoding, clock and data recovery, 
channel bonding (up to 8x) and progranmiable pre-emphasis and equalization. 

1.2 QSFP+ standard 

QSFP+ (Quad Small Form Pluggable +) is an electrical and mechanical standard for point-to- 
point links over copper or optical fibers aimed at high-speed data rate, high-density and low-power 
applications with an aggregated bandwidth up to 40 Gbps per direction [0]. QSFP+ is the latest step 
in the evolution from SFP (Small Form Pluggable) standard. It defines an hot-pluggable transceiver 
with 4 transmit and 4 receive channels which physical layer is based on 100 Ohm AC coupled 
CML differential lines together with copper or fiber optic cables. Different interchangeable cable 
assemblies are now available on the market: passive copper cables for data transmission up to 
5m, active copper (cable assembly with integrated amplifiers) for data transmission up to 15m and 
active optical modules for data transmission up to 100 meters. 
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Figure 1. Altera Stratix IV GX 230 development kit (left) and custom PCI form factor mezzanine connecting 
the FPGA serial transceivers to the QSFP+ connectors (right). 

2. Test system 

The test system is composed of an Altera Stratix IV GX 230 development kit [0], based on an 
Altera Stratix IV FPGA with 230k Logic Elements, 14 Mbit of embedded memory and 24 serial 
transceivers (see figure |T|). 8 serial transceivers are used for the PCI-Express gen 2 interface, 14 are 
routed to two Samtec high-speed connectors [|7[] and the remaining are connected to test points on 
the development kit. 

A PCI form factor custom mezzanine has been designed to be mounted on top of the Altera 
development kit (see figure |T|). This mezzanine mounts two 19 mm high Samtec connectors [EO, 
three QSFP+ connectors and cage assemblies and some SMA test connectors. 12 of the 14 Altera 
high speed channels connected to the Samtec connectors are routed to three QSFP+ connectors 
while the remaining 2 are routed to SMA test points. 

The test instrumentation is composed of a high stability few ps jitter clock generator, a 20 GHz 
sampling scope, an external clock data recovery module and a 500 MHz real time scope. 

3. Test results 

The following tests were performed: signal integrity measurements, recovered clock, latency mea- 
surements and error rate measurements. 

The signal integrity was checked with the high bandwidth sampling scope connected to dedi- 
cated SMA test points on the Altera demo board, immediately at the output of the FPGA transceivers 
(see figure 0). The signal integrity was also checked on the mezzanine card immediately after the 
Samtec high-speed tower connector and also after one meter of QSFP+ cable (see figure @). 

Recovered clock stability was checked transmitting a pseudorandom data stream organized in 
128 bit wide words ^ over 1 meter copper QSFP+ cable and checking the relative phase between 
the input and the output clocks (see figure Recovered clock was found stable and in phase with 
the input clock up to 400 MHz. 

^Each one of the four serial transceivers grouped in a single QSFP+ link carries a 32 bit parallel word per input single 
data rate transceiver clock. The clock of transceiver on line is used for channel bonding and as recovered clock for 
clock stability and latency measurements. 
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Figure 2. Eye diagram at 5 Gbps on the development kit (left) and eye diagram at 3 Gbps on the mezzanine 
card (right) after one Samtec connector, two QSFP+ connectors and 1 m of copper QSFP+ cable (right). 
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Figure 3. Recovered clock stability check at 40 MHz (left) and latency measurement at 40 MHz (right). 

Latency was checked transmitting a pseudorandom sequence over 1 meter QSFP+ copper 
cable and rising a flag every time a fixed test word is transmitted and received by the serializer and 
the deserializer respectively (see figure Transmission system latency was found stable up to 160 
MHz transmitting clock. 

Long runs with single line data rate up to 3 Gbps (12 Gbps aggregated bandwidth on a single 
QSFP+ cable) were performed without any error. 

The proposed link was successfully tested up to 3 Gbps data rate (compared to 8.5 Gbps 
achievable with the Stratix IV embedded transceivers). Above this limit we found some signal 
integrity degradation probably caused by the tower connector between the Altera development 
kit and the test mezzanine. Further investigation is still in progress, however this is perfectly 
reasonable given the reduced bandwidth of the 19 mm high QTH Samtec connector (below 5 GHz). 
We were forced to use this connector in our test mezzanine by the connector already mounted on 
the Altera development kit ^ , higher bandwidth connectors will be used in the production release 
of the communication card for the Lattice QCD parallel processor [@]. 

Characterization of signal integrity (and maximum achievable bandwidth) versus serial trans- 
ceivers pre-emphasis and equalization is still in progress, all presented measurements are still with- 
out any pre-emphasis or receiver equalization. Cables and optical fibers of different lengths will 
also be tested in the near future. 



^Of course smaller and higher bandwidth Samtec QTH connectors are used by Altera to connect the TX and RX side 
of the transceivers with a small slice of PCB. 
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4. Conclusions 



Test results and characterization of a data transmission system based on a last generation FPGA 
and a commercial QSFP+ module were presented. The link was successfully operated up to an 
aggregated bandwidth of 12 Gbps per QSFP+ cable. Further tests are planned with different lengths 
of cables and fiber optics and with the production release of the data transmission board mounting 
the high bandwidth connectors needed to exploit the full bandwidth of the Altera transceivers. 
This data transmission system originally conceived for a commercial PC cluster for lattice QCD 
and other computing intensive numerical algorithm can also be of interest for future particle and 
astroparticle data acquisition systems. 
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